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Abstract 



This paper unifies the development of the cutting plane algorithm for 
mathematical programs and variational inequalities by providing one 
common framework for establishing convergence. Strategies for generating 
cuts are provided for cases in which the algorithm yields easy and difficult 
subproblems. When the subproblem is easy to solve, a line search is added 
and a deep cut is selected to accelerate the algorithm. On the other hand, 
when the subproblem is difficult to solve, the problem is only solved 
approximately during the early iterations. This corresponds to generating 
cuts which are nontangential to the underlying objective function. Moreover, 
in the case of variational inequalities, it is shown further that the subproblem 
can be eliminated entirely from the algorithmic steps, thereby making the 
resulting algorithm especially advantageous. 



I. INTRODUCTION 



Many have studied the cutting plane approach for mathematical 
programs for some time. Cheney and Goldstein (1959) and Kelly (1960) first 
proposed it for convex programs. Dantzig-Wolfe (1960) (see also Dantzig, 
1963) proposed the dual equivalent called generalized linear programming or 
column generation. Zangwill (1969) provided new viewpoints and supplied 
a convergence proof based on algorithmic maps. Magnanti et al. (1976) 
showed that generalized linear programming solves Lagrangian dual 
problems even when standard convexity assumptions do not hold. As a 
standard technique, the cutting plane approach often appears in texts such as 
Bazaraa and Shetty (1979). 

For variational inequalities, references on the cutting plane approach are 
considerably less. Zuhovickii et al. (1969) (see also Auslender, 1976) were the 
first to apply it to variational inequalities. Nguyen and Dupuis (1984) 
proposed an acceleration idea and prove convergence using Zangwill's 
algorithmic maps. 

It is well known (see, e.g., Auslender, 1976, Hearn et al., 1984 and Nguyen 
and Dupuis, 1984) that mathematical programs (MPs) and variational 
inequalities (Vis) are related and possess many similar properties. In fact, 
many algorithms for VI problems are based on MP algorithms (see. Marker 
and Pang, 1990). In spite of this relationship, the developments of the cutting 
plane algorithm appear different for these two areas. However, one can unify 
the developments by addressing MPs and Vis in a common framework. In 
this paper, we provide one such framework with which the cutting plane 
algorithm can be derived for both MPs and Vis. This framework is then used 
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to analyze existing schemes for accelerating the cutting plane algorithm. The 
analysis results in a common argument for proving convergence for all 
acceleration schemes regardless of the underlying problems and points out 
when each scheme might be effective. 

For MPs, this paper focuses on problems derived from Benders 
decomposition and Lagrangian duality. For Vis, the paper considers those 
with finite dimension. In Section 2, it is shown that these problems can be 
stated as the following maximin problem: 

u>* = max min {f{x) + i/g(Ar)} 
ueU xeX 

where U and X are nonempty and convex subsets of R'" and R", respectively, 
f{x) is a continuous real-valued function define on X,g{x) is a continuous 
vector-valued function mapping R" into R'", and xv* is the maximin value for 
the problem. For convenience, ti§{x) denotes the dot product of vectors ii and 
g{x). The basic cutting plane algorithm is stated in Section 3 with a 
convergence proof. Section 4 utilizes the framework given in Section 3 to 
analyze existing strategies for accelerating the algorithm. Finally, Section 5 
concludes the paper. 
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2. INSTANCES OF MAXIMIN OPTIMIZATION 



The first two instances of the maximin problem were first observed in 
Magnanti and Wong (1981) and they are stated here for completeness. The last 
instance can be found in a slightly different form in Hearn et al. (1984) and 
Nguyen and Dupuis (1984). 

First, consider the following nonlinear program (NLP) 

min f{x) 
s.t. ^(x)<0 
X e X 

where f{x), g{x), and X are as previously defined with the exception that X is 
additionally assumed to be compact. Then, the Lagrangian dual problem of 
the NLP can be stated as 

max min {f{x) + U£(x)} 
u>0 xeX 

where u now represents the dual vector for the constraints defined by g{x). 
This shows that Lagrangian duality leads to the desired maximin problem. 

When f{x) = cx, g{x) = b-Ax, and X = [x: Dx > d, x > 0}. The above NLP 
reduces to the following linear program (LP): 

min cx 
s.t. Ax>b 
Dx>d 
x>0, 

and the Lagrangian dual of this LP is 

max min {cx + u{b - Ax)}. 
w>0 xeX 
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Letting f(x) = cx and g(x) = b-Ax yields the desired maximin problem. 
However, an equivalent method of obtaining the Lagrangian dual of the LP is 
by applying Benders decomposition to the dual of the LP which can be stated 
as 

max ub+vd 
s.t. uA+vD < c 
u,v > 0. 

Partitioning the decision variables {ii,v) gives 

max max {ub + vd: vD < c • iiA). 

u >0 z ’>0 

By dualizing the inner maximization, the above maximin problem can be 
written as 

maxmin {i/l7 + (c - or 

u>0 leX 

max min {cx + u{b - 
«> 0 * eX 

The last maximin problem is the same as the one obtained with Lagrangian 
duality. 

Our last instance of a maximin problem is a variational inequality (VI) 
problem which can be stated as follows: find a vector x* € S such that 

F(x *){u • x*)>0 Wu €S 

where S is a nonempty, convex and compact subset of K" and F{x) is a vector 
function mapping R" into R”. When F{x) is strongly monotone on S, i.e., 5 a > 
0 such that 

(F(x) - F(u)){x - 1<) ^ a I X - w I ^ V at, w € S, 
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then the VI problem can be stated as [see Hearn et al. (1984) and Nguyen and 
Dupuis (1984)] 

max min {F(:r)(x - u)}, or max min {F(:c):r - wF(:c)}. 
u€S x€S mcS xeS 

Letting f{x) = F(x)x and g(x) = -F(x) again yields the desired maximin problem. 
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3. THE CUTTING PLANE ALGORITHM 



In the basic cutting plane algorithm, the optimal value of the inner 
minimization of the maximin problem at a given point u is denoted as L(u), 
i.e., 

[S]: L{u) = min{/(x) + 

leX 



Since L(u) is defined as the pointwise minimum of a set of functions linear in 
u, L(u) must be concave. To motivate the algorithm, the maximin problem is 
restated as 

[P]; 7V* = max w 

s.t. to < f(^) + ug(y^) Vx/ e X 

u € U and xv is unrestricted. 

Problem [P] has an infinite number of constraints of the form 

which are generally referred to as cutting planes or simply cuts. To avoid 
generating all the cuts apriori, the algorithm initially solves an approximation 
of problem [P] which contains only a few cuts and obtain, say (w',u'), as a 
solution. To further refine the approximation and hence obtain a more 
accurate solution, problem [S] is solved with u = u' to produce a solution x' 
which defines a new cut 

w <f{x') + ug{x’). 
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The approximation of problem [P] is updated and resolved with the addition 
of this new cut. Then, the process is repeated until an optimal solutionis 
found. Below we formally state the algorithm for the maximin problem. 

The Cutting Plane (CP) Algorithm 

Step 0: Find a point x° € X. Set k = 1 and go to Step 1. 

Step 1: Solve the ) master problem 

max zv 

s.t. zu < fix') + iig(x‘) i = 0, ..., k-1 
u € U and zu unrestricted. 

Let {zv'^, i/*-') denote an optimal solution and go to Step 2. 

Step 2: Solve the {k^ ) subproblem 

x^ = argmin{/(r) + 

Step 3; lizo^ = L(n^), then {ii^, x^) solves the maximin problem. 

Otherwise (i.e., zv^ > L{ii^)), replace k by k+1 and go to Step 1. 

In Step 1, the dual of the k^^ master problem is the following linear pro- 



k-1 

min £ fix’) 
y=o 






it-1 

s.t. 1 Ttj gix’) 

;=0 


< 0 




it-1 

.1 

1=0 


= 1 






> 0 


< 

. 

II 

O 
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The CP algorithm with the master problem replaced by its dual as stated 
above is generally known as Dantzig-Wolfe decomposition, column genera- 
tion, or generalized linear programming. 

In both Benders decomposition and Lagrangian dual of an NLP, the set X 
is either constructed or chosen to facilitate the solving of the subproblem in 
Step 2 or, equivalently, problem [S]. In many cases, a closed form solution or 
efficient algorithm exists for the subproblem. However, such is not the case 
for Vis. The choice for X is restricted to be S, the feasible region of the VI 
problem. Moreover, the objective function of the subproblem contains the 
term F{x)x which is at least quadratic, unless F(x) is a constant vector. 
Therefore, the subproblem, or problem [S], does not generally admit an easy 
solution and the CP algorithm as stated above would be inefficient, if not 
ineffective, for Vis. In Section 4, a technique to overcome this difficulty is 
discussed. 

Step 3 uses the fact that > L(u^) which is generally known to follow from 
Lagrangian duality. However, the inequality can also be obtained by noting 
that a:i e X for all j and 

= min{/(x) + < min|/(.r^) + < / < A: - 1} = 

So, regardless of how/(x) and g(x) are derived, the inequality holds. 

The following two lemmas are essential for the convergence of the CP 
algorithm. The proofs of both lemmas are similar to those in Dantzig (1963) 
and Magnanti et al. (1976) with the exception that all references to Lagrangian 
duality are eliminated. However, they are stated here for completeness and 
further reference. 
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Lemma 1: At each iteration of the CP algorithm 

(i) >w^ V /c 

(ii) w^>iv* yk 

(iii) If L{ii^) > iv^, then w* = L{ii^) = 

Proof: The first result follows from the fact that the master problem has 
more cuts than the The second follows from the fact that the master 

problem in Step 1 contains only a finite subset of cuts in problem [P]. 

If L{u^) > w^, we have 

u> * < 

< L(i/') = minj/lx) + 

X c X 

< maxmin{/'(x) + «g(x)} = w* 

ueU x€X 

where the first inequality follows from (ii), the second is given in (iii), and the 
last inequality is because of the maximization over the set U. The above series 
of inequalities implies that (iii) holds. □ 

Lemma 2: If X is compact and there exists a subset K £ (i, 2, ...} such that the 
subsequence {u^}keK is convergent, say to the limit point, then 

limw^ = zo* = L(i/°°). 
keK ^ ' 

Proof: Since (zt;^, u^) solves the /c* master problem and the fact that constraints 
in the master problem, with j < k, are always contained in the problem at 
iteration k, we have that 

f[x^) + > w'^ > 10 * for; =0,l,...,[k - 1) (1) 

where the last inequality follows from Lemma 1. Let 
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oo 

w = limzy . 
keK 

Note that w ~ exists because is a monotonically decreasing sequence 

which is bounded below by w*. Taking the limit in equation (1) for keK, we 
obtain 

> ly * for; = 0,l/2,3, ... (2) 



By assumption, g{x) is continuous and X is compact. There must exist a posi- 
tive real number j3 such that | g{x) [ < p for all x e X. Then, 

[f(x^) + - f(x^) - ir^(x'^)| = \{ii^ - »“)^(x'^)| < fi\u'' - ir |. (3) 



Consequently, for any given e > 0, there is a k\ e K such that for all keK and 
k>k\, the last term in (3) is bounded by e. Therefore, 



£ > f(x^) + ll^g(x^) - /(X*^) - W“^(x^) > - £. 



Examining the right inequality, we obtain 

L(u*) - /■(.x‘) + >/(i‘) + ir^(i‘) • e 

or L(„‘)r£>/(x‘)*,r4x‘). 

From (2) and the definition of u>*, 

XV * +£> L^u^j + £ > /(x^j + 2 U>°° > XU*. 

Since e is arbitrary, we can conclude that 
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w* 




]r 

limw 

k€K 



where the middle equality follows from the continuity of L(u). g 



To obtain a solution to the maximin problem, evaluate the function L(u) 
at u°° to produce x°° ,i.e., 

= argmm{/(x) + u“g(x)}. 

Then, it follows from Lemma 2 that solves the maximin problem. 

We now obtain convergence for the three instances of maximin problems. 
For Benders decomposition of an LP, the second condition in Lemma 1 will 
hold after a finite number of iterations because the set X is assumed to be 
compact and can be expressed as a convex combination of a finite number of 

extreme points [see, e.g., Bazaraa et al. (1990)]. For VI problems, both U and X 
are the same as S which is compact. Since e S for all k, {u^}^. must contain a 

convergent subsequence, thereby satisfying the hypothesis of Lemma 2. So, 
the CP algorithm converges for Vis. The convergence for the Lagrangian dual 
of an NLP follows from the theorem below. 



Theorem 3: If there exists an e X such that g{x°) < 0 then there is a 
converging subsequence of . 

Proof: See Fisher and Shapiro (1974). 

This theorem also points out that the selection of the initial solution x° in 
Step 0 of the CP algorithm is critical for the Lagrangian dual case, for it deter- 
mines the convergence of the algorithm. Magnanti et al. (1976) provides a 
procedure to obtain an initial solution satisfying the condition in Theorem 3 if 
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one is not readily available. This procedure is akin to phase one of the two 
phase method in LP. 
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4. STRATEGIES FOR GENERATING CUTS 



The two main steps in the CP algorithm consist of solving the master 
problem and generating a new cut. In this section, we examine existing 
schemes for generating new cuts. 

Solving the subproblem in Step 2 of the CP algorithm is one method of 
generating new cuts. However, it does not distinguish between easy and 
difficult subproblems. When the subproblem is easy to solve, it would be 
advantageous to solve more subproblems in a effort to obtain better cuts, i.e., 
those which may lead to a reduction in the number of master problems to be 
solved. On the other hand, when the subproblem is difficult to solve, it may 
be better to cheaply obtain a legitimate cut, perhaps not necessarily tangential 
to L(u). The two subsections below discuss these two schemes in detail. 

4.1: Easy Subproblems 

Hearn and Lawphongpanich (1989a) viewed the difference of two 
successive iterates of the CP algorithm as a direction, i.e., cl = They 

showed that, when Liu) is differentiable at u^'^, d is an ascent direction. 
Otherwise, the nondifferentiability at implies that the subproblem at the 
iteration has multiple solutions and the ascent property would depend 
on the choice of In any case, this observation suggests the inclusion of a 
line search step. Below is one version of the CP algorithm with line search. 
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The Cutting Plane Algorithm with Line Search (CPLS) 

Step 0; Find a point x° e X. Let = 0 and k = 1. Go to Step 1. 

Step 1: same as before. 

Step 2: same as before. 

Step 3: If ), then i/*) soK^es the maximin problem. Otherwise (i.e., 

> L{ii^)), set 

(i) If = 1; set and A: = ^+1. Go to Step 1. 

(ii) Otherwise {k > 1), let 

fmax = arg max {L(d^'-i + td^)-.Q < t < f^p} 

where tup is the maximum value of t for which + td^ remains 
feasible to U. If fmax ^ then pick any nonzero /*•' e [tmax4]- 
Otherwise, let € [l,fmax]- Set = v^~'^ + t^d^ and solve the 
(sub)problem 

/ = argmin{/(x) + 

X €X 

If is not unique, select y* € so that 

/■(y^) + ^ /(y^) + (4) 

Set = y^' and k = k+1. Go to Step 1. 



Note that CPLS produces two sets of dual iterates, and v^’, where 
denotes the solution of the master problem and is a point along the 
direction d^ = Moreover, the cuts are generated at instead of ii'^. 

In Step 3(ii), the choice of step length, in the direction d^ is inexact to 
allow for inaccuracy in the line search and for heuristic selection of a new cut. 
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However, setting = t^ax places at the maximum point and = 1 places 
at which reduces CPLS to the basic CP algorithm. 

For any allowable choice of the resulting is feasible to the master 
problem and we have the following relationship 

L(u^) < L(v^) < min{ f(x‘) + v^g(x ‘) : i = 0,...,{k-l)} < 

If L(vk) = zpk, then CPLS would discover that {x^, v^) is optimal in the next 
iteration. Otherwise (i.e., L(i>k) < w^), adding the cut 
w < f(xk) + ug(xk) 

makes the point infeasible to the next master problem thereby ensuring 

that the sequence ZL’k decreases monotonically. 

To establish convergence for CPLS, it is assumed that either U is compact 
or L(u) satisfies the following condition 

[A]: lim L{u + td) = 

t -» OO 

for all u € U and for all of its direction of recession, d ^ 0 (see page 61 of 
Rockafellar, 1970). Among the three instances of maximin problems, U is 

compact for VI problems because U = S and S is compact by the standing 
assumption in Section 2. For optimization problems, U = {»,• > 0 : / = 1, . . . , m} 

implies that every component of its direction of recession must be 
nonnegative. Then, the requirement (in Theorem 3) that there exists an € X 
such that < 0 further implies that condition [A] holds. To verify this, 
note that for any u e U and any direction of recession d * 0 

L(u + td) = min {f(x) + (u + td)g(x)} < f(x°) + ug(x°) + tdg(x°). 
xeX 

Since d > 0 and^x°) < 0, dg(x°) < 0 and taking the limit as t approaches gives 



15 



limL(ii + fd)</(x“) + u^(x“)+ lim tdg{x")= 

f OO f CO 

Under the assumption that either U is compact or condition [A] holds, the 
value for in Step 3 of CPLS must be bounded. This in turn implies that v 

f; 

is also bounded whenever u is bounded because 

= Iv’' < \v’" -M^I + ll/^1 < max{l, 

Then, the convergence of CPLS follows from the lemma below. 

Lemma 4: If X is compact and there exists an index set K Q {1, 2, ...) such that 
the subsequence converges to w“then there exists a subset 

X' c X such that converges to and 

lim iL’^ = xo* = Lfy°°) 
keK' ^ ' 

Proof: Using the same argument as in Lemma 2, it can be shown that 

+ u” > xu°° > zv* V; = 0,l,2,--- (5) 

and 

Using inequality (4) in Step 3(ii), the right expression is bounded above as 
follows. 

L(y^) = f^x ^ ) + + ii^g[x^) ^ /(^^) ^ 

or j + £ > f{^x^) + u^g^x^^ > 

From (5), we have 

XV* +e> £ -/(^^) '^^‘^g{x’^) - f{^^) 
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Since [u’^]keK converges, i/ must be bounded for all k £ K. It follows from 
the preceding discussion that v for allk & K must be bounded as well. Thus, 
there must exist a subset K' c K such that the subsequence converges 

to v°°. So, taking the limit of the above inequalities with respect to k € K', we 
have 



w* 




> L 



limy 

UeK’ 



k 



+ £ >W*. 



Since £ is arbitrary, u;* = L(y“). 



□ 



Given Lemma 4, obtaining the x component for the maximin solution 
corresponding to v°° and establishing the convergence of the three instances 
are the same as in the basic CP algorithm. 

At the end of Step 3(ii), originally defined in Step 2 is replaced with y^. 
So, now defines a new cut for the {k+1 master problem. Moreover, y^ is 
not arbitrarily selected when alternate optima exist. In fact. Step 3 requires 
that y^ satisfies inequality (4) and the theorem below ensures that such y^' 
exists. 



Theorem 5: In Step 3 of CPLS, there exists a y^ which solves 

L(y^ ) = min[/(x) + 

X ex 

and satisfies 

/(y^) + u^giyf^) <f{y^) + y^^(y^) = l{v ^ ) 
where is part of the solution, to the k^ master problem. 

Proof: Assume that > 1. Denote Then, = B + 

liiaX rri3 X • 

(1-/J) u for some ^ G (0,1) and by concavity of L(u) 
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Uv'') > p ) + (l-p) L(u) > L(u). 

Let X(v ) denote the (compact) set of solutions to the subproblem. To obtain a 
contradiction, assume that 

f(x) + g(x) > f(x) + g(x) for all at e X(v^) 

Then 

(u - V ) g(x) > 0 for all x e X(v^) 

which implies that 

min { (i/*^ -v^) g(x) ; x e X(v^)} > 0 

k k k 

Thus, the directional derivative of L(z’) at v in the direction (u - v ) is 

k k k 

positive, i.e., fu - r ) is an ascent direction. However, y is a point on the line 

connecting to and, by the concavity of L(u), moving toward must 

k k 

decrease its value. This contradicts the statement that (u - ) is an ascent 

direction. (The case for < 1 is proved similarly.) □ 

Figure 1 illustrates cuts which satisfy or are 'allowed' by inequality (4). In 
this case, there are an infinite number of allowable cuts and any of them 
would make CPLS converges. 

As in Magnanti and Wong (1981), we say that the cut, u’ < f{x')+ug{x'), 
dominates or is stronger than the cut, w <f{x") + u^(x")if 

fix') +ug[x') < fix") +ugix") Yu ell 

with a strict inequality for at least one ii. We call a cut pareto optimal if no 
cut dominates it. Since a cut is determined by x G X, we also say that x' 
dominates x" (or x' is stronger than x”) if the associated cut is stronger, and we 
say that x is pareto optimal if the corresponding cut is pareto optimal. 
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It is interesting to note that every allowable cut in Figure 1 is pareto 
optimal. However, this is not always true. In general, there are allowable cuts 
which are dominated. Theorem 6 shows how to modify Step 3(ii) of CPLS to 
generate a pareto optimal cut. 




Figure 1. Allowable Cuts for Step 3 of CPLS 
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Theorem 6: Let X{v^) denote the solution set of the following (sub)problem. 

min{/(x) 

X €X 

If uniquely solves 

min k(x) + 
xex(y*'') 

then is pareto optimal. 

Proof: Assume the contrary that y*-' is not pareto optimal, i.e., there exists an 
X € X and x such that 

f{x) + u^x) < /(y^) + »^(y^’) Vn € a. (6) 

In particular, setting u = in (6) we obtain 

/(.t)+t^^^(x)</(y^)+z>'^g(y'^). (7) 

Since y^' € X{v^), 

f[y^] + ^ ( 8 ) 

Combining (7) and (8) produces the following equality 

f[y^] + ^’^g(y'^) = fix) + v’^g(x) 

that is, X € X(u^). However, setting u = u*' in (6) gives 

fix) + u'^gix) < /(y^) + u^^(y^). 

Since X y*-' and y^' uniquely solves the 2nd minimization problem, this last 
inequality is a contradiction. □ 
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If in Theorem 6 is in the relative interior of U , the uniqueness 
assumption for can be dropped and the pareto optimality follows from 
Theorem 1 in Magnanti and Wong (1981). The following theorem shows that 
in Theorem 6 defines an allowable cut. 



Theorem 7: Let y*^ be as defined in Theorem 6. Then 

fif) + ) 



Proof: Assume the contrary that 

fiy^) + >/{y*) + v^g{y^). 



Using the definition of y*-', we have that 



min 

y€X(cA) 



{/(y) + > f[y^]^v^g[y^) 



{/(y) + i/*'^(y)} - f(y^) ■ V^g(y^) > 0. 

yCXIz^*- j 



Since /(y^) + v^g{y^) - L{v^) = f(y) + v'^g(y) for all y £ X{v^), 

[f{y) + u’^giy) - f{y) - g{y)] > 0. 

yeXftAj 



y€X(t;‘^) 

The expression on the left side is the directional derivative of L{v^) in the 
direction (u^-v^), and the inequality implies that u^'-v^ is an ascent direction 
which is impossible for our choice of step size To see this, assume that £ 
[fmax/1]- This implies that is on line joining y"’** and u^, where u"'** = + 
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fmax maximizes L(u) in the direction {u^-v^). So, moving away from lA' 

toward can only decrease the function value, i.e., u^-v^ cannot be an ascent 
direction at tA. The argument is the same for e [l,fmax]- □ 

Theorems 6 and 7 demonstrate that it is possible to select an allowable cut 
which is also pareto optimal. Magnanti and Wong (1981) showed that pareto 
optimal cuts can accelerate the convergence of the cutting plane algorithm, 
particularly for Benders decomposition. Selecting an allowable cut or a 
pareto optimal would require considerable effort in general. Hearn and 
Lawphongpanich (1989a & b) described a heuristic method for selecting an 
allowable cut. The method consists of the following two rules: 

i) ^max ^ ~ ^max ~ 3nd 

ii) If fmax — 1/ set = frnax ^ 

where e is a small positive number. Figure 2 illustrates how these rules select 
a cut. The function L(u) is nondifferentiable at v,„ax and t„,ax < 1 in Figures 
2(a) and (b). So, letting = t„jax + £ would set to the right oivmax by an e 
amount. However, at the function L(u) is differentiable and the the cut 
generated here is the line tangential to L(u). In Figure 2(a), this e-purturbation 
rule chooses the only nondominated cut. In 2(b), there are an infinite number 
of nondominated cuts and all of which are convex combinations of the two 
'extreme' cuts. In this case, the rule selects one of the extreme cuts. 
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Figure 2. A Heuristic Section of Allowable Cut 



23 



4.2 Difficult Subproblems 



In this situation, one alternative is to solve the subproblem only approxi- 
mately while ensuring that effective cuts are still being generated. In 
Lagrangian relaxation or subgradient optimization (see, e.g., Fisher, 1981 and 
Polyak, 1969), the optimal objective function value is used to define the step 
length calculation which guarantees convergence. For the CP algorithm, the 
maximin value, zv*, analogously defines how close the approximation need to 
be in order to generate an effective cut. In general, the value w* is unknown at 
the start of the algorithm and must be estimated to make the resulting 
algorithm effective. However, zv* = 0 for VI problems (see, e.g., Zuhovickii et 
al. 1969 and Auslender 1976). This allows an approximate subproblem 
solution to be easily obtained. 

Below, we state a version of the basic CP algorithm which solve the sub- 
problem approximately. The name 'nontangential' is due to the fact that, by 
approximately solving the subproblem, the generated cuts are not necessarily 
tangential to L(u). 

The Nontangential Cutting Plane Algorithm (NTCP) 

Step 0: same as before. 

Step 1: same as before. 

Step 2; If = zv*, then there exists an optimal solution (zv^,u^) such that 
L(u*0 = zv*. Otherwise (i.e., zv* < zv^), select e X such that 

f{x^) + u^g{x^) <zv*. (9) 

Set k = ^+1 and go to Step 1. 
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Note that in Step 2 need not be an optimal solution to the subproblem. In 
fact, the (subproblem) objective function value at only need to be 
sufficiently small, i.e., no larger than w*. This insures that {w^,u^) is infeasible 
to the master problem because 

f{x^) + ii^gix^) <w* < w^. 

From this inequality, it can be concluded that is still a monotonically 
decreasing sequence. However, because x^ does not necessarily solve the sub- 
problem, the cut: f(x^) + ii^g{x^) is not necessarily tangent to L{ii) at ii^ (see 
Figure 3). Nevertheless, the following results show that NTCP can still 
converges to the desired solution. 
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Theorem 8: If = w*, then there exists an optimal solution the the master 

problem, such that L{u^) = w*. That is, {x^,u^) solves the 

maximin problem, where 

= min{/'(x)+w*^^(x)}. 

Proof: Assume that the theorem is not true. Then, for every pair {w^,u^) which 
is optimal to the master problem, the following must hold 

< min|/(x^) + = 0,...,(A: - l)j = = w*. (10) 

However, for any pair {zo,ii) feasible, but nonoptimal, to the master prob- 
lem, the following also holds 

L{u) < min|/[.t') + = 0,... ,{k - 1)| < w < xv^ = xv*. (11) 

Combining (10) and (11), we get 

L(u) <xo* for all u £ U 

or max L{ii) < xo*. 

It 6Lf 

This is a contradiction since Tt’* is the maximin value. □ 

Lemma 9: Assume that NTCP generates an infinite sequence If X is com- 
pact and there exists a subset X £ {1, 2, ... } such that the subse- 
quence is convergent, say to the limit point, ii°°, then 

Xi>* = limziJ 
keK 

Proof: Using the same argument as in Lemma 2, it can be shown that for k 
sufficiently large and k € K 
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( 12 ) 



TV* < < f(x^) + If^gix^) ^f(X^) + U^gix'^) + £. 

However, by the choice of x^ in Step 2 of NTCP 

f{x^) + ii^gix^) <TV* 

Combining (12) and (13) yields the following 

TV* < TV^<f{x^) + U'”g{x^) ^f{x^) + U^g{x^) + £<TV* + e. 

Since e is arbitrary, we can conclude that 

TV* = limry^ 
k£K 

To address the convergence of {i/ ton*, where 

TV* = min {f(x)+ii * ^(at)}, 
xeX 

define the following 

i) X°° = } 



ii) 


[x"l - 


the closure of X°° 


iii) 


II 


min {fix') 






ie{0,l {k-\)}'~ 


iv) 


II 


min {/■(x) + w^(a:)} 






xe|x°“| 



From (iii) and (iv), it is clear that 

L“(i/)< L^^\t/)<L^w) ViieU, 
and it can be shown that 

lim L^(iO = L”(iO ViieU. 
k-^ °° 



(13) 

(14) 



□ 
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k 

In other words, [Uii j} is a monotonic sequence of functions which converges 

pointwise to L~(u). Furthermore, note that 

k k 

u = argmax L («). 

u eu 

k M 

If the subsequence {ii converges to u , then it follows frofn Theorem 3.7 of 

Wets (1983) [see also the results in Wets, 1980] that 

u°° = arg max L°°(u), 
u€U 

and (14) also implies that 

w* = L°° (u°°) = lim w^. 

keK 

Lemma 10: Under the same assumptions as in Lemma 9, u* solves the 
following problem: 

max (u). 
u eU 

Proof: Note that 

ry* = min {/(.r) + * ^(.r)} < min {f(x) + u*g{x)} 

< min , {/(.t') + K* 5 (x')} 

< 

where the first inequality follows from the fact that [X”] is a subset of X, the 
second from the fact that [0,l,2,...,{k-l)] is a subset of [X~], and the last from the 

th 

fact that ii* does not necessarily solve the k master problem. 

The above sequence of inequalities can be summarized as follows 

zv* < L°°{u*) < . 

By taking the limit as k ,k € K , and invoking Lemma 9, we have that 
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lT { u*) = w* = max L”(«) 
u eu 



and the proof is complete. □ 



An immediate consequence of Lemma 10 is that if the problem 

max L” (ii) 
u eU 

admits a unique solution, then (i/ must converge to u*. Given Lemma 9 and 

10, the convergence of NTCP for the three instances of maximin problem can 
be established as in the case of the basic CP algorithm except for Benders 
decomposition. To insure that the NTCP algorithm for Benders 
decomposition converges finitely, the chosen in Step 2 of NTCP must also 

be an extreme point of the region X. 

It is interesting to specialize NTCP to VI problems. Because of the 
structure of Vis, the task of selecting an in Step 2 is much easier and the 
stopping rule and the convergence result can both be strengthened. Recall 
that the maximin problem for Vis takes the following form 

w* = max min {F(x).r - F(x)»} 
i/c5 a: eS 

where S is nonempty, convex and compact subset of R” and F{x) is a vector 
valued function mapping R” into R". Moreover, F{x) is further assumed to be 
strongly monotone which implies that there is a unique solution, ii*, to the VI 
problem, i.e., 

F{ii*){x - iC) >0 V X G S. 

Below, we state the version of NTCP algorithm for Vis. 
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The Cutting Plane Algorithm for Variational Inequalities (CPVI) 

Step 0: Let x° £ S. Set k = 1 and go to Step 1. 

Step 1: Solve the (k^) master problem 

max ZD 

s.t. xo < F{x‘)x^ F{x‘)u i = 0, (A--1) 

u e S 

Let {zv^fUk) be an optimal solution and go to Step 2. 

Step 2: Uxv^ = 0, stop and x‘ioi some i € {0, (k-1)} is a solution. Otherwise, 
select x^ £ S such that 

Set k = k+1 and go to Step 1. 

The stopping rule in Step 2 follows from the fact that iv* = 0 for VI 
problems (see, e.g., Zuhovickii et al. 1969 and Auslender 1976). Theorem 11 
and 12 below describe how to obtain the solution to the VI when CPVI 
terminates finitely and when it generates an infinite sequence, respectively. 

Theorem 11: If zu'^ = 0 and F{x’){x’-u'^) = 0, then x* solves the VI problem. 

Proof: Under the strong monotonicity assumption, the solution to the VI 
problem must be unique [see, e.g., Auslender (1976) and Hearn et al. (1984)]. 
Thus, there must be only one x’ such that 

0 = zv’^ = Fix’) ix’ -I/*) 

and for i t j 

0 = zd’^ < Fix') ix'-u’'). 
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Then, the master problem can be reduced to a problem with only one cut, 
i.e., 

0 = = ma:c w 

s.t. u> < F{x’) (x’-ii) 
u € S, 

or 0 = = max f{x^](x^ - n). 

ueS ^ 



However, 



0 = max - ») > F^x^){x^ - u) 



Yu eS 



Multiply through by -1 to obtain 

- .t^) > 0 V w e S. 

That is, xJ solves the VI problem. □ 



Theorem 12: The solution to the VI problem, u*, is a limit point of the set [ X°°]. 

th 

Proof: For each k, let i(k) denote an index of an active cut for the k master 
problem, i.e., i(k) satisfies 

r/ = - F{x^^^^)u^. 

Then, we have 

< (F(;c'U)) . F{u*)){x^^^'^ - u*) 

= - u*) - F{in{x^^’''> - u*) 

< F{x‘^^'>){x^'''^ - It*) 

= 

The first inequality follows from the definition of strong monotonicity and the 
second from the fact that u* solves the VI problem, i.e.. 
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f (i<*)(x • It*) > 0 



Vx €S. 



Since S is compact, there must exist a subset K — {0, 1, 2,. . . } satisfying the 
condition in Lemma 9, thereby havdng the property that 

0 = io*= lim 
keK 

Thus, from the above sequence of inequalities, it must be true that 



lim 

keK 



.m 



- u 




or lim = it*. 
keK 



□ 



i(k) 

From the above theorem, there exists a subsequence of active cuts, {x } , 

fc 

converging to it*. Alternately, Lemma 10 also implies that the sequence {u } 
also converges to u* when it 'strongly' solves the VI problem (see Figure 4), 
i.e., 

F{u*){x • u*) > 0 Vx€S & x*xi*. 

In this case, the problem: max{L°°(i/j : i< e S } admits a unique solution. To 
verify this, assume that u' ^ u* is an alternate solution to the problem. Then 

L~(i/') = min {F{x){x -u')} <F{u*Xu* ■ii’)<0. 

.re|x“| 

where the first inequality follows from the fact that it* £ [ X~] and the second 
from the fact that u* 'strongly' solves the VI problem. However, from Lemma 
10, u* also solves max(L°°(i/j : u e S } and L°°(u*) = w* = 0. Thus, the above 
inequalities is a contradiction since L'”(u*) = C°(u') = 0. Therefore, when u* 
'strongly' solves the VI problem, the entire sequence { u ) must converge to u*, 
since every convergent subsequence of {i/} must converge to a solution of the 
problem max{L~(i/l : w e S } which has a unique solution (see, page 234 of 
Bazaraa and Shetty, 1979). 
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Figure 4: A 'Strong' Solution to a Variational Inequality Problem 

The rate of convergence of CPVI depends in part on the choice for x*-' in 
Step 2. The choice requiring the least effort is due to Zuhovickii et al. (1979); 
they set Under this choice 

F(x^')(x*^ - 11^') = - i/') = 0. 

Nguyen and Dupuis (1984) proposed another choice of x^'. They let x^ be the 
solution to the VI problem over the line segment joining x*‘~^ and denoted 
as , u^]. Thus, x*’ € , t/*- ] and satisfies 

F(x^)(y -x'")>0 V y 

Since 
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F(x*')(i/^ - x'") > 0 



or 



- u'") <0. 



So, Nguyen and Dupuis choice of satisfies the requirement in Step 2. 
Figure 5 illustrates this choice of x'^. In Figure 5(a), if F{x^) is a gradient of a 
function, say,/(x), thenx^' would be the unconstrained minimizer of a one- 
dimensional optimization problem. In Figure 5(b), the 'unconstrained 
minimizer' is to the left of k*-'. So, x must be set equal to However, this 
suggests a generalization of the choice for x^ by Nguyen & Dupuis. 

Let 

a^p = max|a:a > 0 and x^'^ + a(u*^ - x^'^) £ S} 

and set 

k-l ( k k-\\ 

z\,p = .r + a^p[u - X ). 

Then, choose x*^ £ f[ i>up / ] such that 

F(/)(y-/)>0 Vy 

Since £ C[ t>up / ], 

f(x^)(i/'' - x^') >0 or F(x'")(x^ - u'') < 0. 

Thus, the new choice of x*^ also satisfies the requirement in Step 2. Figure 6 
illustrates the new choice of .x^. 
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Figure 5. Nguyen & Dupuis's Choice for 
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Figure 6. The New Choice for x*-' 
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5. CONCLUSION 



This paper addresses the cutting plane algorithm which has been pro- 
posed for problems derived from two different but related areas in operations 
research: optimization and equilibrium problems. In optimization, the cut- 
ting plane algorithm is often used to solve the master problem from Benders 
decomposition or Lagrangian dual of a nonlinear program. For the equi- 
librium problem, the algorithm has been applied only to finite dimensional 
variational inequalities. To unify ideas in the two areas, this paper views 
problems in both areas as maximin problems. This establishes a common 
framework for analyzing and examining convergence properties of various 
schemes for enhancing the cutting plane algorithm. In particular, the analysis 
also leads to several interesting new results. First is the relationship between 
adding a line search step and generating 'deep' or pareto optimal cuts. 
Second are the concept of generating nontangential cuts and the justification 
for solving the subproblem approximately. The last are an alternate proof of 
convergence and a generalization of an existing method for generating cuts in 
the case of variational inequalities. 

It has been well demonstrated that the acceleration strategies discussed 
herein actually reduced the computing time. Hearn and Lawphongpanich 
(1989a & b) compared the CP algorithm against CPLS on 25 Linear and 25 
quadratic integer programming problems. The number of variables in these 
problem ranges from 20 to 100 with number of constraints ranges from 10 to 
50. They concluded that the addition of line search saves cpu time on the 
average 40% for linear problems and 70% for quadratic. The rather large 
saving for the quadratic case is due to the fact that the line search only require 
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two dual function evaluations. In Hearn and Lawphongpanich (1990), the 
same comparison was made on a traffic assignment problem from Goffin 
(1987) with 22 arcs, 14 nodes, and 23 origin-destination pairs. The saving due 
to line search is approximately 50%. 

As for variational inequalities, its subproblem is generally a nonlinear 
program. Thus, by entirely eliminating the subproblem from the steps of the 
CP algorithm, CPVI can only accelerate the basic algorithm. To demonstrate 
that CPVI is competitive with existing methods for variational inequalities, 
Nguyen and Dupuis (1984) compared CPVI with = Z’* against the well- 
regarded Frank-Wolfe algorithm (see, Frank and Wolfe, 1956 and LeBlanc et 
al., 1975) for the traffic assignment problem. They showed that CPVI with x = 
V* compares favorably with Frank-Wolfe on problems with 19 arcs, 11 nodes 
and 4 origin-destination pairs to problems with 2836 arcs, 1052 nodes and 147 

origin-destination pairs. Also shown is the fact that CPVI with x^ = v* is 

k k k 

superior to the one with .t - u . This is not unexpected since setting x = is 

tantamount to performing a line search along the direction t/*' - x^ ^ (see 
Hammond, 1984). 
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